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ABSTRACT 

The use of knowledge in inductive learning is critical for improving the quality of the concept definitions 
generated reducing the number of examples required in order to learn effective concept definitions and 
reducing the computation needed to find good concept definitions. Relevant knowledge may come in 
many forms (such as examples, descriptions, advice, and constraints) and from many sources (such as 
books, teachers, databases, and scientific instruments). How to extract the relevant knowledge from this 
plethora of possibilities, and then to integrate it together so as to appropriately affect the induction process 
is perhaps the key issue at this point in inductive learning. Here we focus on the integration part of this 
problem; that is, how induction algorithms can, and do, utilize a range of extracted knowledge. 
Preliminary work on a transformational framework for defining knowledge-intensive inductive algorithms 
out of reknvdy knowledge-free algorithms is described, as is a more tentative problem-space framework 
that attempts to cover all induction algorithms within a single general approach. These frameworks help 

o organize what is known about current knowledge-intensive induction algorithms, and to point towards 
new algorithms. v 

INTRODUCTION 

Inductive learning is a process whereby a definition of a concept is derived from a set of positive, and 
sometimes negative, examples of the concept. Key issues in inductive learning are the accuracy of the 
resu ting definition - that is, the error rate it yields in classifying new examples - and the resources 

required to generate the definition (in terms of the number of examples and/or the amount of time and 
space needed). 


The single most promising route towards reducing both the error rate and resource usage of inductive 
learning is to utilize whatever additional knowledge is available beyond the examples; that is, to convert 
induction from a weak to a strong method. However, to do this, the relevant knowledge must first be 
extracted from the sources in which it exists, such as books, teachers, databases, and scientific 
instruments This extracted knowledge must then be integrated together for use by the induction process, 
in whatever form is appropriate - examples, descriptions, advice, constraints, or anything else Here we 
focus on the integration task (extraction involves potentially everything from vision to natural language 
understanding, and more). Our goal is to begin the process of deriving principles for how knowledge- 
.ntensive induction algorithms both do now, and can in the future, provide such integration. The hope is 

nrescrini W1 f ^ 3 descn P tlve frame work for organizing existing approaches, as well as a 

prescriptive framework for generating new approaches that go beyond the existing ones. 
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We’ll make this beginning by presenting two partial frameworks for knowledge integration in induction, 
along with implications drawn so far by applying them to four recently proposed knowledge-intensive 
induction algorithms. The focus here is specifically on knowledge integration for induction, rather than 
on the broader issue of knowledge integration in general, in the hope that the extra structure provided by 
the induction problem will lead to more powerful integration strategies than have been proposed for the 
general case. The more developed of the two knowledge-integration frameworks - and thus the one 
emphasized in this paper - is the transformational framework. It describes how knowledge-intensive 
induction algorithms are, and can be, derived by transforming traditional learning methods. The more 
tentative problem-space framework attempts to go beyond the transformational framework to the more 
difficult task of characterizing the fundamental components of all induction algorithms, whether 
knowledge-intensive or not. This framework is covered only briefly here as an intriguing possibility for 
the future. 

THE TRANSFORMATIONAL FRAMEWORK 

An induction algorithm can be viewed abstractly as a black box with one output port for the concept 
description and one or more input ports. A minimal induction algorithm has just one input port, for 
training data, with all other information being hardwired into the algorithm as a fixed bias [6]. The only 
way such an algorithm can use additional knowledge - other than by reprogramming - is to find some 
way of recoding the knowledge as pseudo-examples. For example, Quinlan describes how knowledge 
about type restrictions on the arguments of predicates could conceivably be used indirectly by the FOIL 
algorithm through the generation of pseudo-negative examples that cause FOIL to eliminate candidate 
hypotheses that would violate the type restrictions [8]. 

Most induction algorithms actually do provide some additional input ports that allow explicit provision of 
other types of information; that is, of knowledge beyond what is embodied in the examples. For example: 
the candidate elimination algorithm provides an input port for information concerning the partial ordering 
that defines the initial hypothesis space [4]; FOIL provides an input port for the set of relations that can be 
used in candidate hypotheses [8]; backpropagation provides input ports for learning-rule parameters, the 
network structure, and the initial connection weights [11]; and Bayesian learning algorithms provide an 
input port for prior probabilities [1]. Such ports expand the types of information that can be utilized at 
induction time, but still provide a very limited means for incorporating the full range of knowledge that 
may be available. 

The transformational framework starts with the basic notion of black boxes and ports, as described above, 
and views knowledge-intensive induction algorithms as the composition of a core, usually knowledge- 
lean, algorithm plus a set of transformations. Although not all knowledge-intensive algorithms can be 
viewed in this fashion, when they can, the results can be quite informative. The four knowledge-intensive 
algorithms covered in the next section do all fit this framework, and are based on three distinct core 
algorithms - candidate elimination, FOIL, and backpropagation - and on two general types of 
transformations to these core algorithms: 

• A reformulation transformation modifies the core algorithm so that its ports can handle a 
wider range of inputs, either by generalizing its existing ports or by adding new ones. A 
simple example of a reformulation is the modification of a decision-tree learner to allow it to 
accept a task-specific split criterion from an input port, rather than always using the same 
built-in criterion. A more sophisticated example of a reformulation is IVSM’s derivation 
from the candidate elimination algorithm by converting its examples input port to take a 
more expressive class of inputs (i.e., version spaces) [3]. 

• A preprocessor transformation adds to the core algorithm a preprocessor that takes a form of 
input beyond what can be fed directly into the core algorithm’s input ports, and translates this 
broader input into something that one or more of the core input ports can understand. 
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Quinlan s suggestion of using type constraints to create pseudo-negative examples is exactly 
a proposal for a preprocessor transformation. The preprocessor would have an input port that 
can accept type constraints, and would produce negative examples that can be fed into 
FOIL s existing input port. The combination of FOIL and this preprocessor thus defines a 
' ear " ing a lg°nthm that can take as input not only examples and relations, but also type 

EXAMPLES 

Much recent work on induction algorithms has focused on enhancing their ability to utilize additional 
now e ge unng induction, and thus there are many learning systems we could consider. A full survey 
o sue a gonthms is beyond the scope of this paper, so we will focus here on just four recent knowledge- 

ZZS&S “ provides lhe abili,y lo u,fc EBL ' ,ike domain ,heories ’ plus possibly 

• IVSM has the ability to utilize EBL-like domain theories plus models of bounded 
inconsistency [3]. 

(possibly panial) EBL - Iike domai " «*- . 

’ ^S DEL pJ, a r S a 5 ility . to s P ecif y the hypothesis space via a formal grammar - which can 
dude an EBL-like domain theory - plus some simple ordering information [2], 

* theory^[l 3*] S 3 " eUral netW ° rk alg0rithm that has the abilit y to utilize an EBL-like domain 
The remainder of this section considers these four systems in more detail. 

rF S A M J S t h? ed , OI \ the , candidate-elimination algorithm (CEA). It is derived by a reformulation of the 
CEA so that instead of basing its inner loop on the process of updating a version space with respect to a 
single example, it now updates the version space with respect to a second version space (by intersecting 
the two version spaces). This reformulation generalizes the CEA’s examples input port so that it now 
accep s version spaces. In addition to this core reformulation transformation, IVSM also uses three 
distinct preprocessor transform^ions that are enabled by this reformulated input port. One preprocessor 
allows IVSM to emulate the CEA by taking examples and converting them into version spaces. A second 
preprocessor creates version spaces from combinations of examples and EBL-style domain theories A 
third preprocessor creates version spaces from combinations of examples and a model of bounded 
inconsistency. When IVSM is combined with any one of these preprocessors, it actually yields a new 
induction algorithm: IVSM-CEA, IVSM-EBL, or IVSM-BI. 

E ^ L Is based on FOIL - FOIL uses the information provided on its relations input port to determine 

Tddfn P th ,C nS C °" r making t0 thC CUrTent candidate hypothesis. Essentially, it considers 
adding various relations - as instantiated with a mixture of old and new variables - to the current 

flocan ? ,? Fnrr heS1 f and . uses u an inf ormation-theoretic measure to determine which possibility is 
It of y n k-,v k refortmdates thls possibility-generation strategy in two ways. First, it increases the 
set of possibilities by considering adding combinations of relations in a single step, rather than just 
.ndividua relations. Second, it decreases the set of possibilities by eliminating those tha^ llate 
s ram s on t e arguments of the relations (such as type and uniqueness restrictions). The first 

romhTJt " S f PP r? S the addition of an in P ut port for (possibly partial) EBL-like domain theories; the 
combina ions of relations that occur in the condition sides of these rules form the basis for the relation 

arnnrZ S f P T a reformuIated al gorithm. The second reformulation supports the addition of 
an input port for type and uniqueness constraints on the arguments of the relations that are proposed This 
new port directly supports knowledge that FOIL would have needed a preprocessor to use ? 
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GRENDEL is also based on FOIL. The core transformation made in developing GRENDEL is also a 
reformulation of FOIL’S generation strategy for possible modifications to the current candidate 
hypothesis. However, GRENDEL’ s reformulation is both more radical and more general. GRENDEL 
generates possibilities by consulting generation rules specified in a context-free grammar. This supports 
broadening FOIL’S input port from one that can take a list of relations to one that can handle a list of 
context-free grammar rules. A second smaller reformulation allows the processing of possibilities to be 
selectively deferred, and supports the addition of a second input port to specify this simple ordering 
information. The remainder of the GRENDEL story is much like IVSM. The generalized input port 
facilitates the creation of a number of preprocessors that can accept a variety of types of input. This input 
is then translated into grammar rules that can be fed to this new port. These preprocessors allow 
GRENDEL to accept the kinds of input utilized by (among others) EBL, FOIL, and FOCL, and thus to 
emulate these other algorithms. As with IVSM, there is a base GRENDEL algorithm which takes 
grammars as inputs, and then there are a number of other induction algorithms that are based on 
GRENDEL, such as GRENDEL-EBL and GRENDEL-FOIL, which are derived from it by adding 
specific preprocessors. 

Finally, KBANN is based on backpropagation. It adds a preprocessor that takes as input an EBL-hke 
domain theory, plus a list of environmental features not covered by the theory, and translates this 
knowledge into a form that can be fed into backpropagation’ s initial network topology and initial network 
weights ports. It leaves backpropagation’ s remaining input ports - such as its learning-rule parameters - 
intact. 

ANALYZING INPUT PORTS IN THE TRANSFORMATIONAL FRAMEWORK 

The transformational framework makes it possible to examine knowledge-intensive learners in more 
detail, by studying the set of input ports provided by the resulting algorithms, what kind of knowledge 
they can accept, and what key properties they possess (or fail to possess). Although we are still in the 
process of identifying what the key properties are for input ports, the list already includes at least two that 
seem critical. 

• The additivity of an input port is determined by its ability to accept multiple independent 
fragments of knowledge at that port. Additivity is important because additive ports can serve 
directly as integrators for arbitrary amounts of knowledge of the types that they can accept. 

The prototypical example of an additive input port is the example port in standard induction 
algorithms. It can accept arbitrary amounts of new information, and combine it 
straightforwardly with whatever else the system knows. A classical example of a non- 
additive port is the learning-rate parameter in backpropagation. If more information is 
available, how should it be combined with what is already known? Must the old information 
simply be eliminated, and replaced by the new, or should the two values be averaged, or 
should something else happen? 

For additive ports, the way in which inputs are combined usually depends on their 
interpretation. Examples can be viewed as constraints on the behavior of the concept being 
learned, so they are usually combined via an intersection operator. Other types of 
information might be combined via different operators, such as union or average. 

• The ease of use of an input port is determined by how easy it is to express knowledge in the 
language provided by the port. Bayesian priors are a classic case of a difficult-to-use input 
port, with this difficulty most likely being the single biggest stumbling block in using 
Bayesian approaches to learning. Sometimes preprocessors can be added to make a port 
easier to use; however, the port’s basic ease of use will still affect how easy it is to write the 
preprocessors. A good example of such a preprocessor for Bayesian priors is the use of the 
minimum description length principle, which, while it can be viewed as a Bayesian approach, 
replaces the task of assigning a prior probability to every concept with the arguably simpler 


229 


task of choosing an encoding scheme [9], 

l°'j UStT f th . C f tWO P r °P erties of knowledge-intensive induction methods, as viewed from the 
ansformational framework, we return to the four algorithms discussed above. 

The core IVSM algorithm has two input ports, one for the partial-order information on which the version 

canT M baS6d , 3nd ° ne for a collectlon of version spaces. The partial-order port is additive because it 

“r ^ nUm °l £l f mentS pluS ° rdering re,ations amon g them It is also easy to use but 

relationships ZffV ^ 0f) Candidate - generality 

relationships among them. The version-space port is also an additive port - as with the traditional 

example input port ,, can accept an unbounded set of inputs, and combine them 

' :r ,0n i W, :\ W al ,S a | ready b,OW "' I1S ease of “ X is “®n™:diate between that provided P by 

SIT grenDeS “ aM ^ *5 'T 8 “ Sed f ° r «■“ — *- examples)^ and 

, h "l ag , t * gr ENDEL s grammars at the high end. When IVSM’s preprocessors are considered 

tz&z zsur an of which are addi,ive and reia,iveiy easy ,o use (for ,he “ >■- 

pmp^essors CoS’! It” b - "t ° f ' VSM is that ,here is «*>» its •>"« <H«inct 

p eprocessors couldn t all be used simultaneously. Because they all output version spaces and the 

version-space port is additive, it should be possible to intermingle information based^n examples 
toe exi^on'e"" <«»* effectively creating a new algorithm that subsumes the 

,nput . ports ~ f ° r exa mples, (possibly partial) domain theories, and argument constraints - 
Z atolflea? T W" accep u a : bi,ra,y am0U " tS of know,Kl *' » f «“* ■*»£. input lypTTLy 

are also all easy to use for their intended purposes, but difficult to use for other purposes. Y 

2h^ ND ,5 L S 11 ? pUt P ° rtS acce P t examples, grammars, and ordering information (information about 

h SP3Ce Sh0U,d be T ied first) - Regardl " g ease ° f use ’ th * exampleporthas 
the standard properties, the ordering port is similar to a Bayesian-priors port but likely to be somewhat 

easier to use because it is much less demanding; and .he grammar]™, is relatively easy To use for mos! 

purpo^s. e example and ordenng ports are both additive; however the grammar port is only semi- 

additive, in that the grammars are closed under union, but not under intersection. Thus the additivity of 

°" ,he , way in Which ErammaK ^ If a emmmar is used as as„gg«io„ 
hypotheses are most likely - as when grammars are used to encode a domain theory - then 

£ZZ « £££ C ° mbined Wi ' h 0 “ ni0n r. when grammar abused* 

, , he h yP° thesi s space, it is impossible to generate a separate grammar for each constraint 

spaces) 6 " gratC thC constraints b y intersecting the grammars (as IVSM would intersect its version 

KBANN s three input ports accept examples, domain theories, and environmental features. The examples 

dmcTlt m ut f”' ?? er eXamP ' e i P ° n - * " addi,ive and easy '° f ” intended pu^se Tm 
or o er purposes). The domain theory port is additive and easy to use The 

— P ° rt ^ Uke * he CXampIeS P ° rt> being additive a nd easy to use for very limited 
IMPLICATIONS OF THE TRANSFORMATIONAL FRAMEWORK 

Pulling back up now from these detailed analyses to look at the picture more globally, several general 
implications can be discerned. The first implication is that multiple pieces of kludge can be 
com ined in three distinct fashions. The first approach feeds the knowledge into multiple of the core 

exTmD^KBS 011 ! 8, “f ^ ^ StrUCtUre ° f the C ° re algorithm to P erfo ™ the integration. For 

example, KBANN integrates a domain theory with examples by feeding the domain theory to the core 
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network topology and weight ports, while feeding examples directly to the core examples port. The core 
algorithm - that is, backpropagation - then combines this knowledge during its normal processing. The 
second approach utilizes a multi-ported preprocessor that integrates the knowledge provided to its input 
ports in the process of generating input for the core algorithm. One example is GRENDEL-FOCL’s 
emulation of FOCL via a preprocessor that combines knowledge from all of FOCL’s input ports (except 
for the examples port) in the process of converting this knowledge into a single grammar for use by 
GRENDEL. A second example is IVSM-EBL’s use of a preprocessor to integrate knowledge from its 
examples and domain-theory ports in the process of generating version spaces for the core IVSM 
algorithm. The third integration approach is to utilize an additive port that can integrate across multiple 
pieces of knowledge sent to a single port. IVSM is a good example of this, as its version-space port is an 
effective additive input port. 

The second implication is that the insights underlying different knowledge-intensive algorithms can often 
be transferred or combined in useful ways. In cases where two knowledge-intensive algorithms are based 
on the same core algorithm, and where they have transformed the core algorithm in different ways, it 
should be possible to combine many of the transformations without a great deal of difficulty. For 
example, GRENDEL’ s generalization of FOIL’s relations port to accept grammars could be combined 
with FOCL’s techniques for pruning hypotheses using typing constraints. It would be an interesting 
question to see whether this approach would have any advantages over using a preprocessor to 
incorporate all of FOCL’s knowledge into a GRENDEL grammar, as in GRENDEL-FOCL. 

In cases where the core algorithms are different, transfer of a more abstract sort can still occur. For 
example, IVSM’s additivity based on version-space intersection leads to asking whether GRENDEL’ s 
grammars could support a comparable operation: the answer is no, since context-free grammars are not 
closed under intersection. This also suggests the new research topic of modifying GRENDEL so that it is 
more additive. For example, since the intersection of a context-free language and a regular language is a 
context-free language, it might be possible to create a new version of GRENDEL that has an additive port 
for regular languages in addition to the existing (non-additive) port for context-free languages. 

The third implication is that additional effort would be usefully spent looking at how the two general 
classes of transformations could be applied to further aspects of existing algorithms, both those 
considered here as well as others. 

BEYOND THE TRANSFORMATIONAL FRAMEWORK 

The transformational framework is somewhat unsatisfying for several reasons: it does not apply to all 
knowledge-intensive learners; it does not apply to knowledge-weak learners (which actually do achieve 
some forms of knowledge- integration even in simply being able to accept varying numbers of examples 
and learn from them); and it doesn’t say much about how to merge the insights across knowledge- 
intensive algorithms that have different core algorithms. Our continuing work attempts to go beyond the 
transformational framework by developing a problem space framework that attempts to identify the core 
functionalities that underly all induction algorithms, and then to understand how all of the knowledge 
utilized by a learner - examples, domain theories, etc. - is integrated together via its mapping on to 
these functionalities. In terms of the transformational framework, the goal here can be expressed as 
finding a single black box and set of input ports that conceptually lie at the heart of all induction 
algorithms. 

The problem-space framework is organized around the concept of the space of candidate hypotheses, 
thereby continuing the existing line of analyses that have viewed induction as search [12; 5; 10]. In this 
framework the role of knowledge is first off to specify, constrain, and order the elements - that is, the 
states - of this space. In the four algorithms we have focused on here, specification of the states in the 
space occurs rather directly via GRENDEL’ s grammar port, FOCL’s relations port, and IVSM’s partial- 
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order port. Constrains on the set of states considered are provided by IVSM's version snaces and 
FOCLs argument constraints. Ordering information about the states is provided by GMNDEL’s 
ordenng port and GRENDEL's and FOCL’s examples ports (though rather indirectly! thrrartteir 

ty^Tof ' knotSe m G a RENDF| H ° WeVer ’ J° ne .° f ‘ he foUr !ystems allows dlr cct statement of fit three 
’ a- i ■ ® GRENDEL comes the closest, though it requires all constraints to he ctatpH 

indirectly in terms of what can be generated via the grammar. KBANN is the furthest away as it cannot 

fnfot ? ,reCt s : ate 7 n 7 f of *** types, of knowledge. It does howevTacS some such 

featureO 10n fT in ° T - example ’ * ts domain theory (plus information about additional domain 

d6termineS Wh3t 0311 ^ bC ^ thC h ~ by determining 11 ^” 

The remaining use of knowledge in this framework is to provide method-specific knowledge about how 

t ZvV S T 6 ° f hyP ° theSeS - IVSM " 3t ° ne eXtreme ’ in that makes no use of su h fnowLdge 
it always maintains a representation of all hypotheses that are consistent with all of the knowledge 

"7 S H° f • F ° C u L ’ GRENDEL ’ and KBANN a » Uti,ize 8-edy search algori hms FOCL u" s ft 

Z e Hmtare tjdlteTd? 10 T Ca " dida,e ChmgeS a * ea " h “»• '•-rgumcui-co^.ram, 
Lalrrr.PFmF; !;,. ! I rT P 0 " ,0 order ,|K candid ““ (via its information-iheoretic 
, - t NDRL “ ,he dela,led Structure of its grammar ntles to generate the candidates at each 

thev W ° 8r f m r marS that generate the same terminal language could lead to different greedy searches if 
hey are specified ,n terms of different sets of rules. I, also uses the information from its orde^port as 
irs cut at ordenng the candidates, and then its examples port to complete the ordering (again via its 

KBANN uses its examples port to determine the direction in which to 
<Via - its ■«**-■ <° Ermine the 

Although the problem-space framework is still in a very preliminary stage of development one insiaht 

h^te r ETL e hke y d “ a ' ysis . is ,ha '- th0 “8 h a " four <* *he knowledge-intensive algorithms s.udfed 
here use EBL-like domain theories, they use them in three qualitatively different wavs Twr. of th„ 

a gonthms - KBANN and IVSM - trust their domain theories enough to use them to directly affect the 

space of candidate hypotheses, though they do this in different fashions. KBANN uses the domain theorv 

specify the initial space of candidate hypotheses (that is, the network structure). In contrast IVSM us 7s 

the domain theory along with examples) to constrain the space of candidate hypotheses Thai was earl e 

SdemM provided to its partial-order port. FOCL distrusts its domain theory 

the searrt/f h tu • ect on y he search strategy; that is, the domain theory is used only to order 

tW bm iZoab^ m S r’ "" neVCr t0 P TV he SPaCC - 11 thUS gCtS l6SS COnstraint from its domain 
ry, also able to recover more gracefully if the theory is wrong. GRENDEL’s treatment of the 

e d ri" 5; t hOW , "l e domai "' h “^ <- *» converted fnlo a grammar! GRENDEL can 
t 1 e strate §y* in which the theory determines the search space or a FOCL-like 

Hkpf gy, H in K WhlC V h ! the0iy 0rders the search space - In GRENDEL-FOCL, the variant of GRENDEL 
discussed above, the domain theory orders the search space. 

tahTroad^^dee^r 6 Pr ° b ' en>SpaCe framework - lhe d “ved from i, should (hopefully) ge, 

CONCLUSION 

We have begun the process of understanding knowledge-intensive induction algorithms by presenting a 
transformational framework for creating knowledge-intensive methods from knowledge-we^ meTods 
using the framework to analyze four recent algorithms, and deriving from Ze Zyts glneS 
imp lotions about the integration of knowledge in induction. We also described a more preliminary 
problem-space framework that attempts to identify the core functionalities of any learning method an^ 
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how various learning methods are created by mapping out how knowledge sources can be used to define 
these functionalities. 


Beyond what has already been described, one fundamental insight revealed by these two frameworks, and 
the analyses they yield of existing knowledge-intensive learners, is a path towards simple yet powerful 
knowledge-intensive induction algorithms. First, additive ports need to be developed that provide broad 
languages for the basic functionalities of specifying, constraining and ordering hypothesis spaces. 
Ideally, such ports and languages should combine, for example, the best aspects of IVSM’s version spaces 
and GRENDEL’s grammars, yet still cover all of these basic functionalities. Second, comparable ports 
need to be developed to allow knowledge to be used in whatever search method is chosen. For greedy 
methods, this tends to be knowledge about proposing, constraining, and ordering the options at each step. 
Third, a range of preprocessors need to be created that can translate a wide variety of forms of knowledge 
into these ports. Ultimately this leads to a direct concern about knowledge extraction, as the 
preprocessors get closer and closer to the prime sources of knowledge (such as books), and thus raises a 
variety of additional issues about how and when knowledge is extracted. Ultimately the hope is to 
complete these two frameworks, fuse them into a single more comprehensive framework, analyze the full 
space of existing knowledge-intensive induction algorithms, and use the resulting insights to build one or 
more new algorithms that go significantly beyond the existing ones. 
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